Skip to content

Alternative: render from_markdown via markdown-it-py#61

Open
MattFisher wants to merge 9 commits into
ma2za:mainfrom
MattFisher:markdown-it-from-markdown
Open

Alternative: render from_markdown via markdown-it-py#61
MattFisher wants to merge 9 commits into
ma2za:mainfrom
MattFisher:markdown-it-from-markdown

Conversation

@MattFisher

@MattFisher MattFisher commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this is

An alternative implementation of Post.from_markdown() that delegates parsing to markdown-it-py (a CommonMark parser) plus the standard footnote plugin, with a small renderer (mdrender.py) that maps the syntax tree onto Substack's node schema. Node construction is centralised in a new nodes.py module so the (undocumented) schema lives in one place.

Why I'm opening it

First off — this approach wasn't discussed beforehand, and I realise it's a bigger change than a typical PR, so I'm putting it up purely for your consideration; entirely your call on whether it's a direction you want to take.

I started out adding footnote support to the existing hand-rolled parser (#56). As that grew — footnotes, fenced/inline-code edge cases, multi-paragraph definitions — it started to feel like we were re-implementing a Markdown parser. So I prototyped this alternative to see how it compared, and it turned out significantly simpler:

  • from_markdown() drops from a ~270-line hand-rolled parser to a few lines delegating to the renderer (net post.py ≈ −215 lines).
  • Footnotes come essentially for free from the footnote plugin (including multi-paragraph definitions), rather than via bespoke pre-parse text extraction.
  • A real CommonMark parser brings correctness for free (nested structures, edge cases) and removes the fragile overlapping-regex inline parsing.

Trade-offs (flagging honestly)

  • Adds two runtime dependencies: markdown-it-py and mdit-py-plugins (both widely used and well maintained). This is the main thing to weigh.
  • Two intentional, CommonMark-correct behaviour changes vs the old parser (tests updated to match, with comments):
    • Consecutive > lines are one paragraph; blank > lines split paragraphs (standard CommonMark). The old parser made one paragraph per line.
    • Footnote definitions that are never referenced are dropped (not appended to the end).
  • parse_inline() / tokens_to_text_nodes() remain as public helpers (still used by the manual footnote() builder), but from_markdown() no longer relies on them.

Tests

  • All existing from_markdown/parse_inline tests pass (the two semantic-difference tests above were updated).
  • Added test_from_markdown_features.py with end-to-end coverage of every feature listed in the from_markdown() docstring (headings 1–6, bold/italic/bold+italic/inline-code/strikethrough, links, images, linked images, code blocks with/without language, blockquotes, bullet/ordered lists, horizontal rules, paragraphs).
  • Footnote tests include references/definitions, named labels, numbering, multi-paragraph definitions, and code-span safety. 86 passing (excludes the pre-existing live-API tests that require credentials).

Relationship to #56

This is an alternative to #56 — if you'd prefer this approach, #56 can be closed in its favour; if not, no harm done and #56 stands on its own.

@MattFisher MattFisher mentioned this pull request Jun 26, 2026
Merged
@MattFisher MattFisher changed the title Alternative: render from_markdown via markdown-it-py (simpler, footnotes for free) Alternative: render from_markdown via markdown-it-py Jun 26, 2026
@MattFisher MattFisher force-pushed the markdown-it-from-markdown branch from f257c26 to 97ccc9e Compare June 26, 2026 02:52
@ma2za

ma2za commented Jun 27, 2026

Copy link
Copy Markdown
Owner

Hi, thanks for this. I merged #56 first because it was the lower-risk incremental path for footnotes and kept the existing from_markdown output shape intact.

I like the direction here. Moving Markdown rendering onto markdown-it-py is the cleaner long-term approach, but I don’t want to merge this PR as-is yet because it changes a couple of things beyond the parser swap:

  • poetry install / poetry check currently fails because the new Markdown dependencies were added to pyproject.toml but the lockfile is not in sync.
  • Image rendering changes shape from the existing nested captionedImage -> image2.attrs structure to a flattened captionedImage node. That may be valid, but it is a compatibility change from the current library behavior, so I’d prefer this PR preserve the existing image schema unless we have verified the new one against Substack.

If you want to keep going with this PR, I think the next step is to rebase/merge main now that #56 is merged, regenerate the lockfile, and make the renderer preserve the current draft node shapes while still using markdown-it-py for parsing.

@MattFisher

MattFisher commented Jun 27, 2026 via email

Copy link
Copy Markdown
Contributor Author

MattFisher and others added 2 commits June 27, 2026 20:19
@MattFisher
Render from_markdown via markdown-it-py
f89eb7e 
Replace the hand-rolled Markdown parser in from_markdown() with markdown-it-py
plus the standard footnote plugin, and a small renderer (mdrender.py) that maps
the syntax tree to Substack's node schema. Node construction is centralised in
a new nodes.py module so the schema lives in one place.

Footnotes (including multi-paragraph definitions) come from the footnote plugin.
Adds end-to-end from_markdown feature tests covering every documented feature.

Two intentional, CommonMark-correct behaviour changes vs the old parser:
consecutive '>' lines are one paragraph (blank '>' lines split them), and
unreferenced footnote definitions are dropped rather than appended.
@MattFisher @claude
Fix captionedImage to preserve nested image2.attrs structure
d5a438e 
Restore the schema used by post.py's captioned_image() method:
captionedImage wraps an image2 node whose attrs dict carries src, alt,
href and the remaining layout fields. Update mdrender tests to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@MattFisher MattFisher force-pushed the markdown-it-from-markdown branch from 97ccc9e to ce772b1 Compare June 27, 2026 10:30
@MattFisher @claude
Regenerate poetry.lock with markdown-it-py and mdit-py-plugins
0ec3bd7 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@MattFisher MattFisher force-pushed the markdown-it-from-markdown branch from 5538ee5 to 0ec3bd7 Compare June 27, 2026 10:52
@MattFisher MattFisher changed the title Alternative: render from_markdown via markdown-it-py Render from_markdown via markdown-it-py Jun 27, 2026
@MattFisher @claude
Remove prototype/hedging language from module docstrings
1feff2d 
Strip the PROTOTYPE: prefix and "not wired for production" disclaimer
from nodes.py and mdrender.py now that the direction is approved.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@MattFisher MattFisher changed the title Render from_markdown via markdown-it-py Alternative: render from_markdown via markdown-it-py Jun 27, 2026
@MattFisher
Add end-to-end Markdown round-trip test against the live API
890d31a 
Reads a feature-complete Markdown fixture (headings, inline formatting,
links, captioned + linked images, code blocks with footnote-like text,
blockquotes, lists, horizontal rules, and numeric/named/multi-paragraph
footnotes), posts it as a real draft, retrieves it, and compares the
normalised stored document against a saved golden file.

Auth comes from COOKIES_STRING/COOKIES_PATH or EMAIL+PASSWORD; the test
is skipped when no credentials are configured. Image src values are
normalised since they vary per upload.
@MattFisher
Use a real image URL in the e2e fixture
1e18ff3 
So the generated draft renders an actual image if opened. Stored image2
attrs (and the normalised golden) are unchanged, since Substack keeps the
default dimensions for externally-sourced images in the draft API.
@MattFisher
Cover nested lists in the e2e fixture
d2d85b4 
Add all four parent/child list nestings (bullet>bullet, bullet>number,
number>bullet, number>number) to the fixture and regenerate the golden.
Verified Substack stores the nested structure intact on round-trip.
@MattFisher
Cover local image upload, autolinks, and formatted headings in e2e
8cfeda0 
- Add a locally-sourced image to the fixture so the test exercises the real
  api.get_image() upload path; the test asserts the image was uploaded to a
  remote URL. The fixture references it via a token resolved to a CWD-relative
  path (so get_image's os.path.exists check and the leading-slash handling
  both behave), and from_markdown is now called with api=.
- Add an autolink and a heading with inline formatting.
- Normalise image dimension attrs (height/width/resizeWidth) alongside src,
  since Substack fills those in for uploaded images non-deterministically.
@MattFisher
Reference the local image by its real filename in the e2e fixture
4ba0b54 
Drop the {{LOCAL_IMAGE}} placeholder in favour of a plain relative filename
(local_image.png), matching a real authoring workflow. The test renders from
the fixtures directory so the relative path resolves, as get_image expects.
@MattFisher

MattFisher commented Jun 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

I've updated the poetry lockfile and changed the implementation of captionedImage to match what was done in post.py.

I've also added an e2e test in which I've attempted to capture all the markdown features that are supported, but there may be some combinations I've missed - things like multiple paragraphs within bullets within quoted blocks, etc.

I also realised this implementation currently doesn't support markdown tables, but we could add that feature in a separate PR.

Feel free to push any changes to the PR branch by the way!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MattFisher @ma2za